C^ SPECTRAL RIGIDITY OF THE ELLIPSE 

HAMID HEZARI AND STEVE ZELDITCH 

Abstract. We prove that ellipses are infinitesimally spectrally rigid among C°° domains 
with the symmetries of the ellipse. 
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Q^! An isospectral deformation of a plane domain VLq is a one-parameter family Vt^ of plane 

^ \ domains for which the spectrum of the Euclidean Dirichlet (or Neumann) Laplacian A^ is 

, _. ■ constant (including multiplicities). We say that Vt^ is a C^ curve of C°° plane domains if there 

^ . exists a C^ curve of diffeomorphisms ip^ of a neighborhood of VLq C M? with ipQ = id and with 
f2e = (y9e(f2o). The infinitesimal generator X = -^(^^ is a vector field in a neighborhood of Hq 

P_i I which restricts to a vector field along dQo] we denote by X,y = pu its normal component. 

C/^ ■ With no essential loss of generality we may assume that v^elarjo is a map of the form 

-)— > 



^ 



O 
O 



S 



(1) X G dQo — 7- X + p^^x)^^, 

where p^ E C^{[0,eo],C°°{dD.o)), and we put p{x) = 5p{x) := ^|e=oPe(3;)- An isospectral 
deformation is said to be trivial if Vt^ ~ VLq (up to isometry) for sufficiently small e. A domain 
VLq is said to be spectrally rigid if all isospectral deformations Vt^ are trivial. The variation 



K»- \ is called infinitesimally spectrally rigid if p = for all isospectral deformations. 



In this article, we use the Hadamard variational formula of the wave trace (apparently 
f~i. . for the first time) to study spectral rigidity problems (Theorem [2]). Our main application 

is the infinitesimal spectral rigidity of ellipses among C^ curves of C°° plane domains with 
r^ ' the symmetries of an ellipse. We orient the domains so that the symmetry axes are the x-y 

axes. The symmetry assumption is then that each ip^ is invariant under {x,y) — t- (±x, iy). 



Theorem 1. Suppose that VLq is an ellipse, and that Vt^ is a C^ Dirichlet (or Neumann) 
isospectral deformation of VLq through C°° domains with Z2 x Z2 symmetry. Then X^, = 
/\ ' or equivalently p = 0. 



As discussed in §0.21 and §3.2[ Theorem [T] implies that ellipses admit no isospectral defor- 

_ 1 
mations for which the Taylor expansion of p^ at e = is non-trivial. A function such as e ^ 

for which the Taylor series at e = vanishes is called 'fiat' at e = 0. 

Corollary 1. Suppose that Qq is an ellipse, and that e ^ Q^ is a C°° Dirichlet (or 
Neumann) isospectral deformation through Z2 x Z2 symmetric C°° domains. Then p^ must 
be flat at e = 0. In particular, there exist no non-trivial real analytic curves e ^ Q^ of 
Z2 X Z2 symmetric C°° domains with the spectrum of an ellipse. 

Spectral rigidity of the ellipse has been expected for a long time and is a kind of model 
problem in inverse spectral theory. Ellipses are special since their billiard fiows and maps 
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are completely integrable. It was conjectured by G. D. Birkhoff that the ellipse is the only 
convex smooth plane domain with completely integrable billiards. We cannot assume that 
the deformed domains Q^ have this property, although the results of |Sib2t IZ2] come close 
to showing that they do. The results are somewhat analogous to the spectral rigidity of fiat 
tori or the sphere in the Riemannian setting. 

The main novel step in the proof is the Hadamard variational formula for the wave trace 
(Theorem [2]), which holds for all smooth Euclidean domains f2 C M" satisfying standard 
'cleanliness' assumptions. It is of independent interest and has applications to spectral 
rigidity beyond the setting of ellipses. We therefore present the proof in detail. 

The main advance over prior results is that the domains fi^ are allowed to be C°° rather 
than real analytic. Much less than C°° could be assumed for the domains Jl^, but we do not 
belabor the point. For real analytic domains a length spectral rigidity result for analytic 
domains with the symmetries of the ellipse was proved in |CdV] . The method does not 
apply directly to A-isospectral deformations of ellipses since the length spectrum of the 
ellipse may have multiplicities and the full length spectrum might not be a A-isospectral 
invariant. If it were, then Siburg's results would imply that the marked length spectrum is 
preserved [Sibj ISib2t ISib3] . In |ZH [Z2] it is shown that analytic domains with one symmetry 
are spectrally determined if the length of the minimal bouncing ball orbit and one iterate 
is a A-isospectral invariant. The prior results on A-isospectral deformations that we are 
aware of are contained in the articles |GMt IPTt IPT2] and concern deformations of boundary 
conditions. To our knowledge, the only prior results on A-isospectral deformations of the 
domain are contained in [MMJ. Marvizi-Melrose |MM] introduce new spectral invariants 
and prove certain rigidity results, but they do not apparently settle the case of the ellipse 
(see also jAj IA2j for further attempts to apply them to the ellipse). It would be desirable 
to remove the symmetry assumption (to the extent possible), but symmetry seems quite 
necessary for our argument. Further discussion of prior results can be found in the earlier 
arXiv posting of this article. 

0.1. Hadamard variation of the wave trace. We now state a general result on the 
variation of the wave trace on a domain with boundary under variations of the boundary. 
To state the result, we need some notation. We denote by 

(2) EB{t) = cos [t^/^A^], resp. ^^(t) = ^^^^-&^53. 

the even (resp. odd) wave operators of a domain Q with boundary conditions B. We recall 
that Esit) has a distribution trace as a tempered distribution on M. That is, Eb{p) = 
f^p(t)EB(t)dt is of trace class for any p G C^(M); we refer to |GM2t IPS] for background. 

The Poisson relation of a manifold with boundary gives a precise description of the singu- 
larities of this distribution trace in terms of periodic transversal reflecting rays of the billiard 
flow, or equivalently periodic points of the billiard map. For the definitions of 'billiard map', 
'clean', 'transversal reflecting rays' etc. we refer to |GMl[GM2l IPS] . A periodic point of the 
billiard map /3 : B*dQ — ?■ B*dQ on the unit ball bundle B*dQ of the boundary corresponds 
to a billiard trajectory, i.e an orbit of the billiard flow $* on S*Q. We define the 'length' of 
the periodic orbit of /3 to be the length of the corresponding billiard trajectory in 5**^2. Note 
that the 'period' of a periodic point of (3 is ambiguous since it could refer to this length or to 
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the power of /3. We also denote by Lsp{Q) the length spectrum of Q, i.e. the set of lengths 
of closed billiard trajectories. The perimeter of Q is denoted by \dQ\. 

In the following deformation theorem, the boundary conditions are fixed during the de- 
formation and we therefore do not include them in the notation. 

Theorem 2. Let Qq gM."' be a C°° Euclidean domain with the property that the fixed point 
sets of the billiard map are clean. Then, for any C^ variation of Qq through C°° domains 
Q^, the variation of the wave traces 6Tr cos (t\/—A), with Dirichlet (or Neumann) boundary 
conditions is a classical co-normal distribution for t ^ m\dVLQ\ (m G Zj with singularities 
contained in Lsp{Qo). For each T e Lsp^fio) for which the set Ft of periodic points of the 
billiard map (5 of length T is a d-dimensional clean fixed point set consisting of transverse 
reflecting rays, there exist non-zero constants Cy independent of p such that, near T, the 
leading order singularity is 

STr cos(tv^^A) ~--3?{( ^ Cr I P li dpr) {t -T + i^Y^'i] , 

modulo lower order singularities. The sum is over the connected components T of F^. (Here 
6 = ^|e=o- See (d^j for the definition of •yi). 

Here, the function 71 on B*dQ is defined in (!28|) and appeared earlier in |HZ] . The densities 
dpr on the fixed point sets of /3 and its powers are the canonical densities defined in Lemma 
4.2 of |DG] . and further discussed in |GMt IPTt IPT2] . The constants Cr are explicit and 
depend on the boundary conditions. We suppress the exact formulae since we do not need 
them, but their definition is reviewed in the course of the proof. 

To clarify the dimensional issues, we note that there are four closely related definitions 
of the set of closed billiard trajectories (or closed orbits of the billiard map). The first is 
the fixed point set of the billiard flow ^^ at time T in T*Q. The second is the set of unit 
vectors in the fixed point set. The third is the fixed point set of the billiard flow restricted to 
Tq^Q, the set of covectors with foot points at the boundary. The fourth is the set of periodic 
points of the billiard map /3 on B*dQ of length T, where as above the length is defined by 
the length of the corresponding billiard trajectory. The dimension d refers to the dimension 
of the latter. In the case of the ellipse, for instance, d = 1; the periodic points of a given 
length form invariant curves for /3. 

To prove Theorem |21 we use the Hadamard variational formula for the Green's kernel to 
give an exact formula for the wave trace variation (Lemma [T]). We then prove that it is a 
classical conormal distribution and calculate its principal symbol. 

It is verified in |GM] that the ellipse satisfies the cleanliness assumptions. We then have. 

Corollary 2. For any C^ variation of an ellipse through C°° domains Qe, the leading order 
singularity of the wave trace variation is, 

STr cos(tv/=A) --Igf^K ^ Cr f p li dpr) (t-T + zO)"!}, 

modulo lower order singularities, where the sum is over the connected components T of the 
set Ft of periodic points of {3 (and its powers) of length T. 
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0.2. Flatness issues. We now discuss an apparently new flatness issue in isospectral 
deformations. The rather technical assumption that fl^ is a C^ family of C°° domains rather 
than a C°° family in the e variable is made to deal with a somewhat neglected and obscure 
point about isospectral deformations. Isospectral deformations are curves in the 'manifold' 
of domains. The curve might be a non-trivial C°° family in e but the flrst derivative p might 
vanish at e = 0. Thus, inflnitesimal spectral rigidity is at least apparently weaker than 
spectral rigidity. We impose the C^ regularity to allow us to reparameterize the family and 
show that the flrst derivative of any C^ re-parametrization must be zero. This is not the 
primary focus of Theorem [H but with no additional effort the proof extends to the C^ case. 
This flatness issue does not seem to have arisen before in inverse spectral theory, even 
when the main conclusions are derived from inflnitesimal rigidity. The main reason is that 
flrst order perturbation theory very often requires analytic perturbations (i.e. analyticity in 
the deformation parameter e), and so most (if not all) prior results on isospectral deforma- 
tions assume that the deformation is real analytic. But our proof is based on Hadamard's 
variational formula, which is valid for C^ perturbations of domains and so we can study this 
more general situation. Further, the prior spectral rigidity results (e.g. [_GK] ) are proved for 
an open set of domains and metrics and therefore flatness at all points implies triviality of 
the deformations. We are only deforming the one-parameter family of ellipses and therefore 
cannot eliminate flat isospectral deformations by that kind of argument. We also note that 
there could exist continuous but non-differentiable isospectral deformations. 



0.3. Pitfalls and complications. The route taken in the proof of Theorem [H and the 

flatness issues just discussed, reflect certain technical issues that arise in the inverse problem. 

First is the issue of multiplicities in the eigenvalue spectrum or in the length spectrum. The 
multiplicities of the A-eigenvalues of the ellipse (for either Dirichlet or Neumann boundary 
conditions) appear to be almost completely unknown. If a sufficiently large portion of the 
eigenvalue spectrum were simple (i.e. of multiplicity one), one could simplify the proof of 
Theorem [T] by working directly with the eigenf unctions and their semi-classical limits (as in 
the flrst arXiv posting of this article). The dual multiplicity of the length spectrum is also 
largely unknown for the ellipse. Without length spectral simplicity one cannot work with 
the wave trace invariants. Our proof relies on the observation in |GMj that the multiplicities 
have to be one (modulo the symmetry) for periodic orbits that creep close enough to the 
boundary. 

Second is the issue of cleanliness. Theorem |2] and Corollary [2] would apply to any of the 
deformed domains fi^ if the flxed points sets were known to be clean. One could then use 
the conclusion of Corollary H] to rule out flat isospectral deformations. However, we do not 
know that the flxed point sets are clean for the deformed domains even though we do know 
that they have the same wave trace singularities as the ellipse. Equality of the wave traces 
for isospectral deformations of ellipses shows that the periodic points of (3 of Q^ can never be 
non-degenerate. Hence the deformations are very non-generic. It is plausible that equality 
of wave traces forces the sets of periodic points to be clean invariant curves of dimension 
one. But we do not know how to prove this kind of inverse result at this time. 
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1. HADAMARD variational formula for WAVE TRACES 

In this section we consider the Dirichlet (resp. Neumann) eigenvalue problems for a one 
parameter family of smooth Euclidean domains Vt^ C M", 

r ~^B.^,{e) = \]{e)^,{t) in a, 

(3) 

[ B,m,{e) = 0, 

where the boundary condition B^ could be B^^j{e) = ^^^(e)!^^^ (Dirichlet) or d^^'^j{e)\Qfi^ 
(Neumann). Here, A|(e) are the eigenvalues of — A^,, enumerated in order and with multi- 
plicity, and d^^ is the interior unit normal to Q^- We do not assume that ^E'j(e) are smooth 
in e. We now review the Hadamard variational formula for the variation of Green's kernels, 
and adapt the formula to give the variation of the (regularized) trace of the wave kernel. 
Our references are P Eiil IFTYI ^ |FU] . 

We further denote by dq the surface measure on the boundary dfi of a domain Q, and by 
ru = u\qq the trace operator. We further denote by r^u = d^u\gQ the analogous Cauchy 
data trace for the Dirichlet problem. We simplify the notation for the following boundary 
traces K^{q', q) E V'{dVt x dVt) of a Schwartz kernel K{x, y) G V'iW' x W) (or more precisely 
a distribution defined in a neighborhood of dVL x dVt): 

( {r,,r,N,^,N,^K){q',q), Dirichlet, 

(4) K\<^,q) = l 

I i^^'^^rg:rgK){q',q) + (rg>rgA^K){q',q), Neumann. 

Here, the subscripts q', q refer to the variable involved in the differentiating or restricting. 
According to convenience, we also indicate this by subscripting with indices 1, 2 referring to 
the first, resp. second, variable in the kernel. For instance, j^K(q',q) = -S-Kiq'^q). We 

also use the notations du and ^ interchangeably to refer to the inward normal derivative. 
Also, A^,^ is any smooth vector field in Vt extending v. 

We are principally interested in K{x,y) = SB(t,x,y). In the Dirichlet, resp. Neumann, 
case then we have, 

{r!^r^SD{t,q',q), resp. 
VjV^rg/rg5'Ar(t, g', q) + ^r, A S'Ar(t, g', q). 
Lemma 1. The variation of the wave trace with boundary conditions B is given by, 



5TrEB{t) = -i I S'^{t,q,q)p{q)dq. 
^ J ana 

We summarize by writing. 



5TrEB{t) = -^-Trdn,pS'j,. 

We prove the Lemma by relating the variation of the wave trace to the known variational 
formula for the Green's function (resolvent kernel). We now review the latter. 
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1.1. Hadamard variational formulae. In the Dirichlet case, the classical Hadamard vari- 
ational formulae states that, under a sufficiently smooth deformation Q^, 

f d d 

(6) 6Gd{X, x,y) = - / -— Gd(A, x, g)-— Gd(A, q, y)p{q) dq. 

In the Neumann case, 

(7) 

6GN{\x,y)= / VlGN{X,x,q)-V'^GN{X,q,y)p{q)dq-X'^ GN{X,x,q)GN{X,q,y)p{q) dq. 

JdUo JdUo 

Above, the subscript refers to the variable with respect to which the derivative is taken and 
V"^ denotes the derivative with respect to the unit tangent vector. 

We briefly review the proof of the Hadamard variational formula to clarify the deflnition 
of 6G{X, X, y) and of the other kernels. Following |Pee] . we write the inhomogeneous problem 

(-A + X^)u = /, in n; 

u = (resp. di,u = 0) on dQ 
in terms of the energy integral 

E{u, v) = J^ Vm ■ Vvdx + X^ f^ uvdx 

= f^v{—A + X'^)udx + fQ^vd^udq, 
where d^ is the outer unit normal. The inhomogeneous problem is to solve 



E{u,v) = / fvdx, 
Jn 

where u is a smooth test function which vanishes to order one (resp. 0) on dQ for the Dirichlet 
(resp. Neumann) problem. We denote the energy density by e{u,v) = Vu ■ Vv + X'^uv. 

We now vary the problems over a one-parameter family of domains. As mentioned above, 
we use one-parameter families of diffeomorphisms ip^ of a neighborhood of Qq C M" to deflne 
the one-parameter families Q^ = '/'^(^o) of domains. We assume ip^ to be a G^ curve of 
diffeomorphisms with ipo = id. 

The variational derivative of the solution is deflned as follows: Let u^ G H'^{Q^). Then 
iflu^ e H^i^Qo). Put X = -^l^^Qip^, and put 

n d ^ . 

de 
Assume that 6xu G H^{Qq) and that u G H'^^^{Qq). Then ii exists and 6xu = ii + Xu. 
Further, let f be a test function on VLq and use ^~^*v as a test function on Vt^. Now re-write 
the boundary problems as 



e(M, (v9, yv)dx= I fe{{^^ )*v)dx. 
Changing variables, one pulls back the equation to Qq as 

ee{<flue,v)<f*edx = / {if*J'^)vifldx , whcrc e^{u,v) = !fl{e{!f~'^*u,(f~'^*v)). 
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Assuming that u,9xu G H'^{Q.q) and that u E H'^^^{Vlo), so that 9xu = -^(flu^\^=Q exists as 
a hmit in H^{Qq), we have (by the computations of |Peej (8) and (10)) that 

Lo ^(-^ + ^^)^ ^^ = Ino f^ ^^ + lano f^P ^^ + hn^ (^^ ' ^^ - >?uv)pdq 

(8) ( —\^ Jg^ uvpdq Dirichlet, 

[^ Neumann 

To obtain ©-([T]), at least formally, one puts Ue{ipe{x)) = GB,ei^, '^t{x),y),v{x) = Gb,o{^, y, x), 
and ip*fe = 6y{x). Thus, 6GB{X,x,y) = j^\^=oGBA^,<^e{x),y). Assuming y e Of (the inte- 
rior), then y E Q^ for sufficiently small e and one easily verifies that ([H]) implies (l6])-(I7j). The 
Green's kernel depends on e as smoothly as the coefficients of operator A^ on Qq defined by 
the pulled back energy form. Indeed, the resolvent is an analytic function of the Laplacian. 

1.2. Proof of Lemma [H Rather than the Green's function, we are interested in the 
Hadamard variational formula for the wave kernels EB{t), SB{t) (E]), or more precisely, for 
their distribution traces. In fact, by definition of the distribution trace, we only need the 
variational formula for traces of variations 6 J^e~^^^ilj{t)EB{t)dt of integrals of these ker- 
nels against test functions tp G C^(M), which are simpler because the Schwartz kernels are 
smooth. 

We derive the Hadamard variational formulae for wave traces from that of the Green's 
function by using the identities, 

/■OO 7 

(9) i\Rb{\) = J e-'''EB{t)dt, -SB{t) = EB{t) 

integrating by parts and using the finite propagation speed of 5*5 (t) to eliminate the boundary 
contributions at t = 0, oo. It follows that 



(10) 



/"OO 

Rj^{\) = / e-'^' SB{t)dt. 
Jo 



As mentioned above, to obtain the variational formula for the singularity expansion of the 
wave trace, we only need variational formula for the smooth kernels 

(11) J^^P{t)e-'^' EB{t)dt = J^ifiRB{p)i^ip - A)c?/x, 

where Rb{p) = {—^b ~/^^)~^ is the resolvent of — A^. Here we assume that ip is supported 
in M+ since in the wave trace we localize its support to the length of a closed geodesic. In 
the Dirichlet case, it follows that 

(12) 5 I ^p{t)e-'^' EB{t)dt = 5 [ ifxRB{fi)^ip - \)dfi. 

Jr Jm. 
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We then derive variational formulae for wave traces. In the Dirichlet case, it follows from 
(IT2|) and (ED that 

S J^ijj{t)e~'^^ ED(t)dt = -ij^fitp{fi- \) fg^du^GD{fJ',x,q)du^GD{fi,q,y)p{q)dqdfi 

= -Jr ir e~*''V(^ - A) Jg^ d^^Eoit, X, q)d^,GD{f^, g, y)p{q)dqdfidt 

= -Jr ir ir (i~'^^*^*'H{p - A) /g^ d^^Eoit, X, q)d^,SD{t', q, y)p{q)dqdfidtdt' 

= - ir ir e-'^^'^''H{t + t') Jg^ d,,En{t, X, q)d,,SD{t', g, y)p{q)dqdtdt' 

= - ir Ian e"*^^^(r) {£ d^^Eoir - t', x, q)d^,SD{t', q, y)dt') p{q)dTdq. 

The inner integral is the same if we change the argument of Eo to t' and that of Sd to t — t' . 
We then average the two, set x = y, integrate over fl and use the angle addition formula for 
sin to obtain 

(13) 5Tr [ ^{t)e-''''ED{t)dt = -\ j tij{t)e-'^' [ d^.d^^Snit, q, q)p{q)dqdt. 

Jr ^ Jr Jdn 

This is the real part of wave trace variational formula stated in the Lemma in the Dirichlet 
case, i.e. the variational formula for 6 TrEo{t). The proof in the Neumann case is similar 
and left to the reader. 

This concludes the proof of Lemma [TJ To navigate the formulae, we also give a derivation 
based on the Hadamard variational formulae for eigenvalues. When A^(0) is a simple eigen- 
value (i.e. of multiplicity one), then Hadamard's variational formula for Dirichlet eigenvalues 
of Euclidean domains states that 



5(A|)= / {d.^j\9n,rp{q)dq, 



lano 

where "^j is an L^ normalized eigenfunction for the eigenvalue A|(0) and dq is the induced 
surface measure. See |G]. The same comparison shows that if the eigenvalue {A^(0)}^^ 
is multiple and if {^'j^ki^)}^=i is the perturbed set of eigenvalues, then 

m(A,{0)) m{Xj{0)) 

(14) 6 Y^ Xlk= Yl / (^'^^ hk\d^of Kq) dQ = I d^id^^'^x,io){q,q)p{q)dq, 

where {'^j^kYI^=i is an orthonormal basis for the eigenspace of the multiple eigenvalue 
A^(0) and Ilx^(o){x,y) is its spectral projections kernel. Since 6Xj = -^^, by ( 1T4|) we have 

S Tr Es{t) = 6 E cos(tA,,.) = -t E, ( ES^^"^ ^(AiJ) '^^$^ = ^ Jen, d.,d^.Sn{t, g, q)p{q)dq. 

Hence Lemma [H follows in the Dirichlet case. 

There exist similar Hadamard variational formulae in the Neumann case. When the eigen- 
value is simple, we have 

(15) S{X'^) = I {\vl{^UM)? - A?(o)(^,bno(g))^) m dq. 
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For a multiple eigenvalue we sum over the expressions over an orthonormal basis of the 
eigenspace. The result does not depend on a choice of orthonormal basis. Similar computa- 
tion using ( !T5|) follows to show Lemma [1] for the Neumann case. 

2. Proof of Theorem [2] 

We now study the singularity expansion of 5Tr cos{t\/—AB) and prove Theorem [2l At 
first sight, one could do this in two ways: by taking the variation of the spectral side of the 
formula, or by taking the variation of the singularity expansion. It seems simpler and clearer 
to do the former since we do not know how the invariant tori of the integrable elliptical 
billiard deform under an isospectral deformation. In this section we will drop the subscript 
in VLq. 

The variational formula for 5Tr cos{t^J—/S.B) is given in Lemma [H In the Dirichlet case, 
by ©-dSD, 

(16) Trao. p S'^j^ = n^. A* p {rir2N^^N^^SD{t,x,y)), 

where N^ is any smooth vector field in Vt extending z/, and where the subscripts indicate the 
variables on which the operator acts. In the Neumann case by (jl])-®, 

(17) Tran p S'}, = n, A*p ((Vf V^rirs - nr^ A,)5jv(t, x, y)) . 

Here, A : dQ — )■ dQ x dQ is the diagonal embedding q — )■ (g, q) and n^, (the pushforward of 
the natural projection vr : dQ x M — )■ R) is the integration over the fibers with respect to the 
surface measure dq. The duplication in notation between the Laplacian and the diagonal 
is regrettable, but both are standard and should not cause confusion. Since SB(t,x,y) is 
microlocally a Fourier integral operator near the transversal periodic reflecting rays of F^, 
it will follow from ( 1T6|) that the trace is locally a Fourier integral distribution near t = T. 

We are assuming that the set of periodic points of the billiard map corresponding to 
space-time billiard trajectories of length T G Lsp{Q) is a submanifold Ft of B*dQ. We 
thus fix T G Lsp{Q) consisting only of periodic reflecting rays, i.e. we assume T ^ m\dQ\ 
{\dQ\ being the perimeter) for m G Z. In order to study the singularity of the boundary 
trace near a component Fj- of the fixed point set, we construct a pseudo-differential cutoff 
Xt = XT{t,Dt,q,Dq) G \E'°(M x dQ) whose complete symbol XT(t,T,q,() has the form 
Xriq, ^) with xriy, C) supported in a small neighborhood of the fixed point set Ft C B*dVt, 
equals one in a smaller neighborhood, and in particular vanishes in a neighborhood of the 
glancing directions in S*dVt = d{B*dQ). Since the symbol of xt is independent of t we will 
instead use XT{F>t, q, Dq). We may assume that the support of the cutoff is invariant under 
the billiard map f3. Therefore we need to study the operator 

(18) vr,A* p xt{Du g', /^,')Xt(A, g, Dq)Sl{t, g', g), 

and compute its symbol. To do this we first study the operators r and Ssit) and review 
their basic properties. Next we study the composition 

Xt(A, g', Dq,)xT{Dt, q, Dq)S's{t, q' , q) 

and compute its symbol. Finally in Lemma[7]we take composition with 7r*A* p and calculate 
the symbol of flTSj) . 
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2.1. FIOs and their symbol. We recall that the principal symbol cr/ of a Fourier integral 
distribution 

1= [ e'^("'^)a(x,^)rf^, /G/'"(M,A^), 

of order m is defined in terms of the parametrization 

i^:C^ = {(x, 6) : de^ = 0} ^ (x, 4<^) e A^ C T*M 



of the associated Lagrangian A^. It is a half density on A<^ given by o"/ = ('■(p),,(aoMc^h) 
where oq is the leading term of the classical symbol a G S"^~^i~~{M x M^), n =dimM and 

dc 



is the Gelfand-Leray form on Cy, where c is a system of coordinates on Cy,. For notation and 
background we refer to [iHoj. When I{x, y) G I^{X x Y, A) is the kernel of an FIO it is very 
standard to use the symplectic form ux — ojy on X x F and define 

i^ : C^ = {(x, y, 6) : de(p = 0} -> (x, d^ip, y, -dyip) G A<^ C T*X x T*Y. 

We will call A^ the canonical relation of I{x,y). 

2.2. The restriction operator r as an FIO. The restriction r to the boundary satisfies, 
r G I^{dQ X R^jFgn), with the canonical relation 

(19) Ton = {(g,C,g,0 e T*dn X T,*^M'^;e|T,an = C}- 
The adjoint then satisfies r* G /4(]R" x dQ,r*Q^), where 

r^f, = {(g,e,g,C) e T^*^^" x T*9fi;e|T,an = C}- 

Here, TJ^^M" is the set of co-vectors to M" with footpoint on dQ. We parameterize Tg^ (TT^ 
by Tq^{Q), the inward pointing covectors, using the Lagrange immersion 

(20) Lrg^{q,0 = il,^\T,{dn),q,0- 

To prove these statements, we introduce Fermi normal coordinates (g, x„) along dQ, i.e. 
X = expg(x„z/q) where z/^ is the interior unit normal at q. Let ^ = (C^^n) £ ^(g^,,)^" denote 
the corresponding symplectically dual fiber coordinates. In these coordinates, the kernel of 
r is given by 

(21) r(g, [q', x'J) = cj e'^'^-'' '<^-'<^" d^^dC . 



The phase (p{q, {q',x'^), (C^n)) = {q ^ I'X) ~ ^'n^n is non-degenerate and its critical set is 
C^ = {(g, g', x^, Cn, C); ?' = ?, x'^ = 0, }. The Lagrange map i^ : (g, g, 0, ^„, C) -^ (g, C, Q, C, ^n) 
embeds C^ — ?■ T*dQ x T*R" and maps onto Fg^. The adjoint kernel has the form K*{x, g) = 
K{q, x) and therefore has a similar oscillatory integral representation. It is clear from fl?Tl) 
that the order of r as an FIO is j. Also, in the parametrization ( 120|) . the principal symbol 

of r is (Tr = |(ig Ad( A (i^„| 2 . 
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2.3. Background on parametrices for ^^(i). We first review the Fourier integral de- 
scription of EB{t), Ssit) niicrolocally near transversal reflecting rays. This is partly for 
the sake of completeness, but mainly because we need to compute their principal symbols 
(and related ones) along the boundary. Although the principal symbols are calculated in the 
interior in |GM2l IPS] and elsewhere (see Proposition 5.1 of |GM2] . section 6 of [MMj and 
section 6 of [PS]), the results do not seem to be stated along the boundary (i.e. the symbols 
are not calculated at the boundary). The statements we need are contained in Theorem 3.1 
of |Ch2j (and its proof), and we largely follow its presentation. 

We need to calculate the canonical relation and principal symbol of the wave group, its 
derivatives and their restrictions to the boundary. We begin by recalling that the propagation 
of singularities theorem for the mixed Cauchy-Dirichlet (or Neumann) problem for the wave 
equation states that the wave front set of the wave kernel satisfies, 

WF{SB{t,x,y)) c\Ja±, 

± 

where A± = {{t,T,x,^,y,r]) : {x,^) = ^\y,i]), r = ±|?7|y} C T*{R x fi x fi) is the graph 
of the generalized (broken) geodesic flow, i.e. the billiard flow $*. For background we refer 
to [UM2l[P5l[Ch2] and to [HoJ (Vol. Ill, Theorem 23.1.4 and Vol. IV, Proposition 29.3.2). 
For the application to spectral rigidity, we only need a microlocal description of wave kernels 
away from the glancing set, i.e. in the hyperbolic set microlocally near periodic transversal 
reflecting rays. In these regions, there exists a microlocal parametrix due to Chazarain |Ch2j . 
which is more fully analyzed in |GM2t IPS] and applied to the ellipse in |GM] . 

The microlocal parametrices for Eb and Sb are constructed in the ambient space M x M" x 
f2. Since Eb = ^Sb it suffices to consider the latter. Then there exists a Fourier integral 
(Lagrangian) distribution, 

oo 

SB{t,x,y)= Y^ Sj{t,x,y), with Sj e I--^-\R x W x Q,ri) 

j=~oo 

which microlocally approximates SB{t,x,y) modulo a smooth kernel near a transversal re- 
flecting ray. The sum is locally finite hence well-defined. The canonical relation of Sb is 
contained in a union 

r= U ri cT*(MxM" X n) 

of canonical relations P^ corresponding to the graph of the broken geodesic fiow with j 
refiections. Notice we let j ^1^ which is different from |Ch2] where j goes from to oo and 
where the two graphs P^. and P^"' are combined. 

To describe Pj_, we introduce some useful notation from |Ch2] with a slight adjustment. 
We have two Hamiltonian flows g'^^ corresponding to the Hamiltonians ±|?7|. For {y.,ri) in 
T*VL we deflne the flrst impact times with the boundary, 

tl{y,r^) = inf{t > : Tig^\y,r^) G d^}, 

t±\y,r]) = sup{t < : 7rg^\y,r]) G dQ}. 

The impact times are related by t^^ = —t^- We deflne t''^ inductively for j > res. j < to 
be the time of j-th reflection (i.e. impact with the boundary) for the flow g^^ as t increases 
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resp. decreases from t = 0. Then we put 

Next we define X].{y,r]) to be the reflection of X^{y,ri) at the boundary. For any {q,C,) G 
T*W", q G dQ, the reflection ^ — )■ ^ has the same tangential projection as ^ but opposite 

normal component. Similarly we define X^^{y,ri). Flowing X].{y,ri) (resp. X^^{y,ri)) by 
g"^^ as t increases (resp. decreases) and continuing the same procedure we get t-'^iy^rj) and 
Ai(y, 7]) for all j G Z. We also set T^ = Y.i=i 4 ^^ i > and T^ = Ei=-i 4 fo^ i < 0- 
The canonical graph T-'^ can now be written as 

{{{t,r,g^\y,r]),y,ri): t = ±\r]\y} j = 0, 

{(t,r,^±(*-^±(^'^))Ai(i/,r^),l/,r^) : r = ±|^|,} j G Z,j ^ 0. 

For each j G Z, |J_|_ F ^ is the union of two canonical graphs, which we refer to as its 'branches' 
or 'components' (see figure 3.2 of |GM2] for an illustration). These two branches arise because 
5*5 (t) = rrA— i^^^^~'^^ ~ g-«V-Afl^ jg ^Yie sum of two Fourier integral operators whose 
canonical relations are respectively the graphs of the forward/backward broken geodesic 
flow and which correspond to the two halves r > 0, r < of the characteristic variety 
r^ — Ir^p = of the wave operator. 

At the boundary, we have four modes of propagation: in addition to the two ± branches 
corresponding to r > and r < 0, there are two modes of propagation corresponding to the 
two 'sides' of dQ. To illustrate this we first discuss the simple model of the upper half space. 

2.3.1. Upper half space; a local model for one reflection. Let M" = {(x',x„) G R"^^ x M : 
Xn > 0} be the upper half space. Denote by So{t,x,y) the kernel of /~^ of Euclidean 
M". By the classical method of images. 

Suit, X, y) = So{t, X, y) - S^it, x, y*), 
Snit, X, y) = Soit, x, y) + S'o(t, x, y*) 

where y* G M" is the reflection of y through the boundary R"~^ x {0}. 

The canonical relation associated to S^ and Sc is the union of the canonical relations of 
Sq and of Sq = So(t,x,y*). More precisely by our notation in (!22|) 

WF{SBit,x,y)) crlurluri\ 

Note that this example is asymmetric in past and future: the forward trajectory may intersect 
boundary, but then backward one does not. Also, in this example for j > 1 and j < —1 the 
graphs F^ are empty. 
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2.3.2. Symbol of S sit, x, y) in the interior. In the boundaryless case of |DGj . the half density 
symbol of e* v ~ » is a constant multiple (Maslov factor) of the canonical graph volume 
half density acan = \dt A dy A dri\2 on r+ in the graph parametrization (t,y,ri) — )• r+ = 
{t, \rj\g,g^{y,ri),y,'r]). In the boundary case for Esit) the symbol in the interior is computed 
in Corollary 4.3 of |GM2] as a scalar multiple of the graph half-density. It is a constant 
multiple of the graph half-density 

(23) acan,± = \dt A dy A dri\^ 

in the obvious graph parametrization of F:?^ in (|22|) : the constant equals | in the Neumann 

case and ^^ in the Dirichlet case. However in |GM2] the symbols are not calculated at the 
boundary. 

2.3.3. Symbol of SB{t,x,y) at the boundary. Since we want to restrict kernels and symbols 
to the boundary, we introduce further notation for the subset of the canonical relations lying 
over boundary points. Following jCh2] . we denote by A^. = {{0,T,y,ri,y,ri) : r = ±|?7|y} 
the subset of F^. with t = 0. Under the flow ijj^ of the Hamiltonian r ± |,^|a. on M x M", it 
flows out to the graph F^. (denoted C° in |Ch2] . (2.11)). One then defines A]_ C F^. resp. 
A^^ C F^ as the subset lying over M+ x (9f2 x ^2 resp. ]R_ x Sfi x fi. We then have 

Fi = u^i^L 

teR 

and 

F- = U^i^i\ 

as the flow out under the Euclidean space-time geodesic flow of A\. and ^±^. Thus, along 
the boundary, for t > (resp. t < 0) A^ and A}^ (resp. A^^ and A'^^) both lie in the 
canonical relation of Esit), SB{t)- In a similar way one deflnes A"^ to be the subset of Fj_ 
lying over M+ x dQ x Q and A^ to be its reflection. Then also A"^ U A"^ lies in the canonical 
relation. Similarly one deflnes A-!^ and A-!^ for all j G Z. 

Remark: Since we are interested in the singularity of the trace at t = T > we will only 
consider the graphs F:?^ for j > 0. Regardless of this, because 6 TrEsit) is even in t it has 
the same singularity at in t = L and t = —T. 

The symbols of Esit) and SB{t) are half-densities on the associated canonical relations, 
and therefore are sums of four terms at boundary points, i.e. there is a contribution from 
each of A-'^ and A'^. In the interior, there is only a contribution from the ± components. 

The following Lemma gives formulas for the principal symbol of Sb (and therefore Eb) 

on T-'^ and its restriction to Tqq^ o {^A'^ U A-'^. 

Lemma 2. Let e± be the principal symbol of Sb when restricted to r± = [JjT-'^. Let a^ be 
the principal symbol of the boundary restriction operator r. Then 

1. In the interior, on F-^, up to Maslov factors we have e± = 2t ^can,± = ± 21 1 ^can,± 
in the Dirichlet case, and e± = ^crcan,± = ^^\0'can,± in the Neumann case. 
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2. At the boundary, on Tqq, o A^ = Tgn o A^. we have 
In the Dirichlet case: 

ar o e±(ti, ±r, Ai(y, r/), y, r]) = -a^ o e±(ti, ±r, Ai(2/, r/), y, r^), 
In the Neumann case: 

ar o e±(ti, ±r, Ai(?/, r/), y, r/) = a^ o e±(ti, ±r, Ai(|/, r]),y, rj). 

Proof. These formulas are obtained from the transport equations in [Ch2j, (&o) — (cq) (page 
175). We now sketch the proof. 

The transport equations for the symbols of Eb, Sb determine how they propagate along 
broken geodesies. As in the boundaryless case, the principal symbol has a zero Lie derivative, 
'^H^+iji'^-B = 0, in the interior along geodesies. The important point for us is the rule by 
which they are reflected at the boundary. Let as be the principal symbol of the boundary 
restriction operator B defined in (^ {B = r resp. B = rN when we have Dirichlet resp. 
Neumann boundary condition) and let ctq be the principal symbol of the restriction operator 
to t = 0. Then, 
(24) 
f (bo) : (^ - Ab)Sb ~ =^ (6'o) : £^^e± = 

(co) : 5^B|t=o ~0 ^ (c[,) : ao o e+{0,T,y,r],y,r]) + ao o e^{0, -T,y,r],y,r]) = 

(do) : ft\t=o^B ~5(a;-y) =^ {d'^) : T{ao o e+{0,T,y,r],y,r]) - ao o e_{0, -T,y,r],y,r])) = aj 

(eo) : BSb ~ ^ (e'o) : as o e± = ctb o (e±|^,^) + as o (e±|— ) = 0. 

Here o"/ is the principal symbol of the identity operator. The implication (bo) =^ {b'^) follows 
for example from Theorem 5.3.1 of |DHj . The other implications are obvious. From (cq) and 
(d'o) we get 

i(rooe±){y,r],y,r]) = —ai, on T*n. 

Zt 

But by (6q), the symbol e± is invariant under the flow ip^. and therefore the flrst part of the 
Lemma follows but only on F^. The second part of the Lemma follows from (e'o). The flrst 
term of (cg) is known from the previous transport equations. Hence (e'o) determines the 're- 
flected symbol' at the j-th impact time and impact point. In the Dirichlet case, B is just r the 
restriction to the boundary and so the reflected principal symbol is simply the opposite of the 
direct principal symbol. In the Neumann case, B is the product of the symbol {\\{y, rj), Uy) 
of the inward normal derivative times restriction r. The reflected symbol thus equals the 

direct symbol since the sign is canceled by the sign of the {X±{y,v)y^y) = ~ {^±i.y y v) y ^y) 
factor. Thus, the volume half-density is propagated unchanged in the Neumann case and 
has a sign change at each impact point in the Dirichlet case. It follows that, on T-!^ and after 
j reflections, the Dirichlet wave group symbol is (— l)-* times ^ times the graph half-density 
fl23|l and the Neumann symbol is 27 times the graph half-density. 

D 



C°° SPECTRAL RIGIDITY OF THE ELLIPSE 15 

2.4. xriDt, q' , Dqi)xT{Dt, q, Dq)S'^{t, q' , q) is a Fourier integral operator. 

Lemma 3. We have, 

XT{Dt,q',Dg,)xT{D„q,Dq)S's{t,q',q) G I'^+'-^R x dQ x dQ,Te,^). 
Here, 1^,^ = UjezTa^i, with 

n,± ■■ = {(t, r, q', C, q, C) eT*{Rxdnx dQ) : 3 ^ G T;,M", ^ G T^M" : 

(t, r, q', r, g, e ri, r |T,,aQ = C, eiT,an = C}- 

Proof. We only show the proof in the Dirichlet case. The Neumann case is very similar. 
The kernel XT^Dt, q' , Dqi)xT{,Dt, q, Dq)S'^{t, q' , q) for fixed t is the Schwartz kernel of the 
composition 

(25) XT o{rN)o Soit) o (A^* r*) o Xt ■ L^{dn) -> L^{dn), 

where r* is the adjoint of r : H2(pi) — ). L'^i^dVl). 

To prove the Lemma, we use that r is a Fourier integral operator with a folding canonical 
relation, and that the composition ( 125|) is transversal away from the tangential directions 
to (9f2, where SB{t) fails to be a Fourier integral operator. The cutoff xt removes the part 
of the canonical relation near the fold locus, hence the composition is a standard Fourier 
integral operator. 

By the results cited above in |Ch2l [GM2l IPSi [MM[ ICh2"] . microlocally away from the gliding 
directions, the wave operator Ssit) is a Fourier integral operator associated to the canonical 
relations F^. Since F^ is a union of graphs of canonical transformations, its composition 
with the canonical relation of r^ = vN is automatically transversal. The further composition 
with the canonical relation of r^* is also transversal. Hence, the composition is a Fourier 
integral operator with the composed wave front relation and the orders add. Taking into 
account that we have two boundary derivatives, we need to add | to the order. 

To determine the composite relation, we note that 

$± : M X T^^W' -^ r*R X T*n x T^^Vt, 
(26) 

<^±(t,g,C,en):=(t,±|C + ai,$*(g,C,en),g,C,a) 

parameterizes the graph of the (space-time) billiard flow with initial condition on Tg^'R"'. 
Here, ( G T*dQ and ^„ G N^dQ, the inward pointing (co-)normal bundle. $± is a homoge- 
neous folding map with folds along M x T*dQ (see e.g. |Hoj (volume HI) for background). 
It follows that Soit) o (N*r*)xT is a Fourier integral operator of order one associated to the 
canonical relation 

{(t,±|e|,$*(g,0,g,e|T*an} C T*(R xQx dQ), 

and is a local canonical graph away from the fold singularity along T*dQ. Composing on the 
left by the restriction relation produces a Fourier integral operator with the stated canonical 
relation. The two normal derivatives A^ of course do not change the relation. 

D 
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2.5. Symbol of XTiDt,q',Dqi)xTiDt,q,Dg)S'^(t,q',q). The next step is to compute the 
principal symbols of the operators in Lemma [3l 

To state the result, we need some further notation. We denote points of T^^M" by 
{q,OX,^n) as above, and put r = \/|CF+~^. We note that ^n is determined by (g, C, t) 
by Cn = v^T^ — |(^P, since it is inward pointing. The coordinates g, ( are symplectic, so the 
symplectic form on T*dQ is da = dq Ad(. We then relate the graph of the billiard flow ( l26l) 
with initial and terminal point on the boundary to the billiard map (after j reflections) by 
the formula 

(27) $^^(g,0,C,U = (r/3^(g,-),C(g,C,en)), 

r 



where ^; = rJl - \f3^{q, J)|2. We also put 



(28) 7(g, C, r) = y 1 - ^, and 7i(g, C) = V^^W- 

It is the homogeneous (of degree zero) analogue of the function denoted by 7 in |HZj . 

Further, we parameterize the canonical relation Tq _^_ of Lemma [3] using the billiard map 
(3 and its powers. We define the jth return time T^{q,^) of the billiard trajectory in a 
codirection {q, ^) G T*Q to be the length the j-link billiard trajectory starting at (g, ^) and 

ending at a point ^'^^^"'^^q,^) G Tq^H. It is the same as T|(g, ^). Then we define 
(29) 

L9,,+ : R+ X T*dn ^T*{RxdQx dQ), ls,{t, q, () = {P{q, ^(g, C, r)), r, (r/3^(g, ^)), g, C), 

T 

where 

e(g,c,r) = c+enz/„ icr+ienr = r2. 

The map fl29|) parameterizes F^ ^ of Lemma [31 

Proposition 4. In the coordinates (r, g, () G M+ x T*dQ of ( 129|) . ^/ie principal symbol of 

XT{Dt, g', Dq')xT{Dt, g, D^jS^it, g', g) 

on Fg _,_ zs as follows: 

• in the Dirichlet case: 

^,,+(g, C, r) = Cf+XT(g, -)XT(/3^'(g, -))7^ (g, C, r)7^(r/3^'(g, -), ^))rMg A dC A rfr|^ 

• m the Neumann case: 

a,,4q, C, r) = CJ^+XT(g, -)XT(/3^'(g, -))7~^(g, C, r)7-^ (r/3^'(g, ^), r)) ((C, /3^'(g, ^))-r) |rfgArfCArfr|i 
where C^^ are certain constants (Maslov factors). 

Proof. We only show the computations in the Dirichlet case. The Neumann case is very 
similar and uses ([5]) which will produce an additional factor of t{(, l3^{q, ^)) — r^. 

By Lemma [21 the principal symbol of Ssit) consists of four pieces at the boundary, one 

for each mode A^, ^4^. The symbol for the — mode of propagation is equal to that for the + 
mode of propagation under the time reversal map ^ — )■ —^. Further by part 2 of Lemma [21 
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the symbol at the boundary (adjusted by taking normal derivatives in the Dirichlet case) is 
invariant under the reflection map ^ — )■ ^ at the boundary due to the boundary conditions. 
Hence we only calculate the A-\_ component and use the invariance properties to calculate 
the symbol on the other components. 

We therefore assume that the symbol of Sb is ^ times the graph half-density \dt Adx Ad^l^ 
on r-^. We need to compose this graph half-density on the left by the symbol ^n\dqAd(Ad^n\^ 
oi r^ = r N, and on the right by the symbol Cnl'^l' ^ '^C /\ '^Cl^ ^^ ^^^ adjoint r^* = N*r*. 
Therefore we compute the restriction of the T-'^ component onto Tq _,_ and we remember to 

multiply the symbol by ^nCn = '^'^liQi Ci T)7(T/3''(q', 7), t)) and also by ^ at the end. 

It is simplest to use symbol algebra and pullback formulae to calculate it (see |DG] ). The 
composition is equivalent to the pullback of the symbol under the pullback 

(30) Vq = {lan X ^an)*^^ 

of the canonical relation of the Sb by the canonical inclusion map 

2af7 X iao : K X 9fi X 9fi ^ M X M" X Vt. 

We recall that a map / : X — )■ F is transversal to ly C T*y if df*ri 7^ for any r] E W. If 
/ : X — )■ y is smooth and F C T*Y is Lagrangian, and if / and tt : T*Y — )■ Y are transverse 
then /*r is Lagrangian. Since 

{idn X ianTit, r, <l>*(g, 0, q, = {t, r, <l>*(g, OItoq, q, ^Wan) 

at a point over {igQ x idn)(t, q', q), and since t = |^| 7^ 0, it is clear that ign x ig^ is transversal 
to 71. 

We now claim that on the pullback of F-', using the parametrization (!29l) . 



(31) {ian X ianTldt A dx A d^\^ = ^-^q,C,Th-H'rf3' {q,-),r)\dq A dC A dT\^ , 

T 

where 7 is defined in (1251) . To see this, we use the pullback diagram 

F^' ^^ F — ^ {ian x ian)*P cT*{Rxdnx dn) 

i I TT 4, 

T*{R X R" xn) ^ Ar*{gTaph{tan x tan)) 

Here, F is the fiber product, Af*gTa.ph.{ian x ian) is the co-normal bundle to the graph, and 
the map a : F — )• {ian x ian)*P is the natural projection to the composition (see |DGj ). 
Since the composition is transversal. Da is an isomorphism (loc. cit.). The graph of iau x ian 
is the set {(t, q, q', t, q, q') : (t, g, g') G M x dfl x dfl} and its conormal bundle is (in the Fermi 
normal coordinates), 

Af*{gmph{iany<ian)) = {{t,T,q,C,q',C',t,-T,q,-C + ^n,q',-C + C), 

iqx,^n),iq'x'.o^nnm 

C T*(Rxdnxdnx 
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The half density produced by the pullback diagram takes the exterior tensor product of the 
canonical half density 

\dtAdT Adq AdC A d^^ A (i^^ A dq' AdC\^ 

on J\f*{gT8iph.{isn x ig^)) and 

\dt' A dx A rf^'l i on F C T* (M x W x fi) 



at a point of the fiber product (where the T*(]R x R" x Vt) components are equal) and divides 
by the canonical half density 

\dt' A dr' A dq' A dC,' A dx'^ A d^ A dx' Ad^'\^ 

on the common T*]R x T*]R" x T*Q component. 

Since t' = r, the factors of \dt' A dr' A dq' A d(' A d^'^ A dx' A d^'\'^ cancel in the quotient 
half-density, leaving the half density 



\dt A dq AdC, A d^r. 



lAitXj ^ 



2 



on the composite. The numerator is a half-density on M x TqqW^. We write it more intrin- 
sically in the following Lemma. Note that it explains the first of our two 7 factors. 

Lemma 5. Let $ = $+ be the parametrization ^2B\j and ujt*w^ he the canonical symplectic 

1 t 1 1 

form of r*]R". Then \dt A dq A dC, A d^r]"^ = \ / I 2 1~^ \^*ujt''R"\^ as half- densities on 

V Id +in 

R X T^^R\ 
Proof. We have, 

_ , , (TT a a a \ 

_ in , ( a a a a \ 



2 



since ji^^q.'q.in) = Hg = ^^"2 ^f„ H is the Hamilton vector field of ^f = -J'g^.g^ = 

^n + id')'^ where ■ ■ ■ represent vector fields in the span of ^, ^, -^. Finally, we use that 

(i$* is symplectic linear and that q,Xn,Cj^n are symplectic coordinates. Note that we are 
evaluated the symplectic volume form at the domain point, not the image point. 

D 

Next we take consider the points in the image of $ on R x T^^^M" where x'„ = and take 
the quotient by |(ix^|2, resulting in a half density on F^. The next Lemma explains the 
origin of the second 7 factor. 
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Lemma 6. In the subset F^ C $(]R x T^^R"') where x'^ = and where t = T^ , we have (in 
the parameterizing coordinates ^2§i)), 

\dt A dq A dC A dU\^ ^^ j _i 



,^ ,, |((/3^)V')rfgArfr/Adr|^ 

Proof. By the previous Lemma, it suffices to rewrite 

\dx'J~^ |<I>*wt*M"P 
in the coordinates {T,q,ri) of lsj in fl29|) . We observe that x'^ = $*a;„. Hence 

\dx'J-i \<^*UJT*Rn\'^ = 



$ 



^ LO'j'* 



\dXn\ 

= \{{/3^y-f-^)dq A dC A drl'- 
In the last hne, we use ( 127|) . that 

— - — p = IdgArfC Arf^nl, 

\ClXn\ 

and that /3 is symplectic. Indeed, by fl27|) . 

^^(dg A dC A rf^n) = T{(3^)*idq A d^) A ^*din 



T 



{l3^y{dqAd^)A^*d^T^-\C\' 



= dqAdCA <^*^^^ = {m*T^)dq A dC A dr. 
Note that T{/3^)*{dq Ad^) = dqA c?C|/3J(g,c)- 

Combining Lemma [6] with Lemma [5] completes the proof of (|3T1) and Proposition |H 



D 



n 



2.6. Trace along the boundary: composition with tt* A*. We now take the trace along 
the boundary of this operator. Analogously to |DGt IGMl IMMj . we define A : M x dD. — t- 
M X dQ X dQ to be the diagonal embedding and n^, to be integration over dQ. 

Lemma 7. // the fixed point sets of period T of (3^ are clean for all k and form a submanifold 
Ft of B*dQ of dimension d (with connected components T), then 

7T,A*pXT{Dt,q\D,,)xT{Dt,q,D^)SUt,q\q) e I-^+-^+'--*{R, T*R), 

where 

T*R = U±At,± = U±{(T, ±t) : T E M+}, 
and its principal symbol on Ax± is given by 



c T 2 y/dr, 
where 



c± 
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and c~ = c+ the complex conjugate of c^ . 

Proof. The calculation of the principal symbol of the trace of a Fourier integral operator in 
|DG] is valid for the boundary restriction of the wave kernel, since it only uses that it is 
7r^,A* composed with a Fourier integral kernel with a known symbol and canonical relation. 
Hence we follow the proof closely and refer there for further details. 
As in |GM] . the composition of vr^A* with 

(32) pxADt, q\ D,,)xt{Du q, D,)SUt, q\ q) 

is clean if and only if the fixed point set of [5^ corresponding to periodic orbits of period T 
is clean. When the fixed point set has dimension d in the ball bundle B*dQ, composition 
with tt^kA* adds | to the order (see [DG] , (6.6)). Combining with Lemma [3l we obtain the 
order f + | + 1 - i 

Hence under the cleanliness assumption, it follows that S Tr cost\/— A^ is a Lagrangian 
distribution on M with singularities at t G Lsp{Q). As discussed in |DG] (loc. cit.) for the 
upper/lower half lines A^^-i- in T^M., I2~^4(M.,At,±) consists of multiples of the distribution 



/ r^e±^-(*-^)rfr=(t-T±*0)- 
Jo 



d+2 



The principal symbol of this Fourier integral distribution is r^~ ydr. Therefore to conclude 
the Lemma we only need to compute the coefficients of this symbol in the trace. 

This coefficient is computed in a universal way from the principal symbol of f l32|) computed 
from Proposition m Following the proof of [DG] . the coefficient of r^~ y/dr is 



c^ = ^ C^ / p 7i df^r, 



FcFt 

where Ft is the fixed point set of /3 (and its powers) in B*dQ. The sum is over the connected 
components F of Ft and dfir is the canonical density on the fixed point component F defined 
in Lemma 4.2 of |DGj . We note that the distribution c'^ {t — T + iO)~ ^~ +c~(t — T — iO)~^~ 
is real only if c~ = c^. This completes the proof of the Lemma. 

D 

The Lemma also completes the proof of the Theorem. We close the section with a remark: 

Remark: As a check on the order, we note that for the wave trace in the interior and for 
non-degenerate closed trajectories, the singularities are of order [t — T + iO)~^. When the 
periodic orbits are degenerate and the unit vectors in the fixed point sets have dimension d, 
the singularity increases to order (t — T + iO)~^~2. If we formally take the variation of the 
wave trace, the singularity should increase to order (t — T + iO)^^^2^^, 

In comparison, the boundary trace in the Dirichlet case involves two extra derivatives of 
the wave kernel and composition with (— A)~2. Compared to the interior trace, this adds 
one net derivative and order to the trace singularity. We claim that the restriction to the 
boundary does not further change the order compared to the interior trace. This can be 
seen by considering the method of stationary phase for oscillatory integrals with Bott-Morse 
phase functions, whose non-degenerate critical manifolds are transverse to the boundary. If 
we restrict the integral to the boundary, we do not change the number of phase variables 
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in the integral, but we simultaneously decrease the number of variables by one and the 
dimension of the fixed point set by one. The number of non-degenerate directions stays the 
same. It follows that the singularity order of the variational trace goes up by one overall 
unit compared to the interior trace, consistently with the formal variational calculation. 



3. Case of the ellipse and the proof of Theorem [T] 

In this section we let Qo be an ellipse. In this case, the fixed point sets are clean fixed 
point sets for $* in T*Qq, resp. for /? in B*dQo (See |GMj Proposition 4.3). In fact the fixed 
point sets Ft of (3 in B*dVtQ form a one dimensional manifold. Thus d = 1 and corollary [2] 
follows. 

As is well-known, both the billiard flow and billiard map of the ellipse are completely inte- 
grable. In particular, except for certain exceptional trajectories, the periodic points of period 
T form a Lagrangian tori in 5**^20, and the homogeneous extensions of the Lagrangian tori 
are cones in T*VLq. The exceptions are the two bouncing ball orbits through the major/minor 
axes and the trajectories which intersect the foci or glide along the boundary. The fixed point 
sets of $"^ intersect the co-ball bundle B*dQo of the boundary in the fixed point sets of the 
billiard map /3 : B*dno ^ B*dno (for background we refer to [ESI EMI CM2l iHZl |TZ2] for 
instance). Except for the exceptional orbits, the fixed point sets are real analytic curves. 
For the bouncing ball rays, the associated fixed point sets are non-degenerate fixed points of 
(3. 

Since the final step of the proof uses results of |GM] , we briefly review the description of 

2 2 

the billiard map of the ellipse f^o := ^ + ^ = 1 (with a > 6 > 0) in that article. In the 
interior, there exist for each < Z < b a. caustic set given by a confocal ellipse 

+ T7 = 1 



where e = a — 6, or for — e < Z < by a confocal hyperbola. Let (g, Q be in B*dQ() and let 
(g, ^) in 5**^0 be the unique inward unit normal to boundary that projects to (g, (). The line 
segment {q, r^) will be tangent to a unique confocal ellipse or hyperbola (unless it intersects 
the foci). We then define the function Z{q,() on B*dQo to be the corresponding Z. Then 
Z is a /3-invariant function and its level sets {Z = c} are the invariant curves of /3. The 
invariant Leray form on the level set is denoted duz (see [GMj . (2.17), i.e. the symplectic 
form of B*dQo is dq A d( = dZ A duz- A level set has a rotation number and the periodic 
points live in the level sets with rational rotation number. As it is explained in [GMj (page 
143) the Leray form duz restricted to a connected component F of Ft is a constant multiple 
of the canonical density d^r- 

As mentioned in the introduction, the well-known obstruction to using trace formula 
calculations such as in Proposition [2] is multiplicity in the length spectrum, i.e. existence 
of several connected components of Ft- A higher dimensional component is not itself a 
problem, but there could exist cancellations among terms coming from components with 
different Morse indices, since the coefficients Cr are complex. This problem arose earlier in 
the spectral theory of the ellipse in [GMj . Their key Proposition 4.3 shows that there are is 
a sufficiently large set of lengths T for which Ft has one component up to (g, () — )■ (g, —() 
symmetry. Since it is crucial here as well, we state the relevant part: 
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Proposition 8. (see |GM] . Proposition 4-3): Let Tq = \dQo\. Then for every interval 
[mTo — e, -mTo) for m = 1,2,3, .. . there exist infinitely many periods T G Lsp{Qo) for which 
Ft is the union of two invariant curves which are mapped to each other by {q, () -^ {q, —()■ 



Since for an isospectral deformation 6 Trcos{ty/—A) = 0, we obtain from Proposition [2] 
the following 

Corollary 9. Suppose we have an isospectral deformation of an ellipse Qq with velocity p. 
Then for each T in Proposition\^ for which Ft is the union of two invariant curves Fi and 
T2 which are mapped to each other by {q, () — )■ {q, —() we have 



p 7i duz = 0, j = 1, 2. 

Proof. From Proposition [2] we get 

2 c 

3^{(E^r, / P7irf/ir,)(t-T + zO)-2-i}=0. 

Since p and 71 are invariant under the time reversal map {q, () — )■ (g, — C), the two integrals 
are identical. Also by directly looking at the stationary phase calculations it can be shown 
that the Maslov coefficients Cr^ and Cpj are also the same. Thus the corollary follows. D 

3.1. Abel transform. The remainder of the proof of Theorem [1] is identical to that of 
Theorem 4.5 of [GM] (see also |PT] ) . For the sake of completeness, we sketch the proof. 

Proposition 10. The on/^ Z2XZ2 invariant function p satisfying the equations of Corollary 
\Bis p = 0. 

Proof. First, we may assume p = at the endpoints of the major/minor axes, since the 
deformation preserves the Z2 x Z2 symmetry and we may assume that the deformed bouncing 
ball orbits will not move and are aligned with the original ones. Thus p{±y/a) = p(±a/6) = 0. 
The Leray measure may be explicitly evaluated (see 2.18 in |GM] ). By a change of variables 
with Jacobian J, and using the symmetric properties of p, the integrals become 

(33) A(z) = r ^^pdm. 

io Vi - (ft - Z) 

for an infinite sequence of Z accumulating at b. Since < a < 6, the function A{Z) is 
smooth in Z for Z near b. It vanishes infinitely often in each interval (6 — e,b), hence is flat 
at b. The A;th Taylor coefficient at b is 

nb 

(34) A^''\b) = / p{t) 71 J{t)t-''-Ut = 0. 

J a 

Since the functions t~'^ span a dense subset of C[a, b], it follows that p = 0. 

D 
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3.2. Infinitesimal rigidity and flatness. We now show that infinitesimal rigidity implies 
flatness and prove Corollary [H As mentioned, the Hadamard variational formula is valid for 
any C^ parametrization fla{e) of the domains Q^- For each one we have 6pa{€){x) = 0. 

Assume Pe{x) is not flat at e = and let e^ be the flrst non- vanishing term in the Taylor 
expansion of Pe{x) at e = 0. Then 

,35) ,,.)., Pi!M,,..Q£),.... 

We then reparameterize the family by e — )■ a(e) := efc so that 



Pa{e)[X) 



-e + 0(ei+Vfc). 



By Hadamard's variational formulae we get 5pa{e)ix) = p^^\x) = 0, a contradiction. 
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